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Abstract 

Capturing engineering lessons learned derived from 
past experiences and new technologies, then integrating 
them with technical standards, provides a viable process 
for enhancing engineering capabilities. The 
development of future space missions will require ready 
access, not only to the latest technical standards, but 
also to lessons learned derived from past experiences 
and new technologies. The integration of this 
information such that it is readily accessible by 
engineering and programmatic personnel is a key aspect 
of enabling technology. This paper addresses the 
development of a new and innovative Lessons 
Leamed/Best Practices/Applications Notes-Standards 
Integration System, including experiences with its 
initial implementation as a pilot effort within the NASA 
Technical Standards Program. Included are metrics on 
the Program, feedbacks from users, future plans, and 
key issues that are being addressed to expand the 
System's utility. The objective is the enhancement of 
engineering capabilities on all aspects of systems 
development applicable to the success of future space 
missions. 

Introduction/Background 

Government and commercial aerospace organizations 
in the United States who are either planning or engaged 
in efforts related to the development of new or 
improved manned and un-manned space vehicles are 
benefiting from the use of technical standards and 
lessons learned gained from previous experiences. 
Technical standards are an essential aspect of all 
engineering development efforts, but are of particular 
significance in the aerospace industry. In 1997 NASA 
established an Agencywide Technical Standards 
Program under the direction of the NASA Chief 
Engineer. 


This Material is declared a work of the U.S. 
Government and is not subject to copyright protection 
in the United States. 


The Program, via its Web site http://standards.nasa.gov , 
has undertaken an extensive effort to make available 
technical standards, including relating them to lessons 
learned data sources, for use in the development of the 
Agency’s programs and related engineering activities. 

NASA is currently involved in the development of an 
Orbital Space Plane (OSP), a next generation space 
vehicle designed to provide a crew rescue and crew 
transport capability to and from the International Space 
Station. This effort will benefit from the lessons 
learned from previous vehicle designs, component 
developments, and operations. The success of this 
critical development effort is largely dependent on the 
applicable lessons learned that are identified and 
applied, as well as the application of relevant technical 
standards. Properly applied technical standards and 
lessons learned enhance engineering capabilities. The 
application of standards and lessons learned will reduce 
the cost of the engineering design and increase system 
reliability. 

The roots of all technical standards are buried in lessons 
learned, and the roots of lessons learned extend 
downward into the depths of our experiences. These 
experiences, both positive and negative, form the basis 
of lessons learned data sources. To be significant, 
lessons learned must have a real impact on the 
program’s function or procedure. Lessons learned must 
be screened and validated to assure that they contain 
accurate technical content. In order for the lessons to 
be effective, they must also be applicable and address a 
specific design process or decision that mitigates risk, 
increases safety and reliability, or leads to an improved 
design or process. Once captured, lessons can then be 
integrated with technical standards to form a powerful 
engineering information source to enhance not only 
space vehicle development efforts, but to improve 
organizational engineering capabilities. This paper is 
based on the experiences gained from the 
implementation of the NASA Technical Standards 
Program Web site, and a pilot study on the integration 
of lessons leamed/best practices/application notes 
with technical standards. 
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Discussion 

The search for relevant lessons learned information can 
be challenging because most lessons learned are not 
written in a standardized format and are disseminated in 
various mediums. They are documented in technical 
papers and memos, professional journals, and 
databases, and can be found in numerous forms. 
Locating lessons relating to particular areas of technical 
expertise or technical disciplines is equally as 
challenging. To overcome this obstacle, the simple 
notion of integrating lessons learned with technical 
standards has proven to be a very viable means by 
which the engineering community can locate and infuse 
these lessons into the design and development of 
specific programs and projects. 

Examples of lessons learned data sources incorporated 
into the NASA Technical Standards Program Web site 
include: 

• NASA/Headquarters- Lessons Learned 
Information System. 

• NASA/Glenn Research Center- Frequently 
Asked Questions on Failures. 

• NASA/Kennedy Space Center- Cryogenic 
Transfer Mechanical Design. 

• NASA/Goddard Space Flight Center- Systems 
Engineering Lessons Learned. 


• AIAA Satellite Mission Operations Best 
Practices. 

• NASA/Langley Research Center- Lessons for 
Software Systems. 

With the lessons learned - standards integration effort 
comes the problem of locating lessons focused on a 
specific technical category. To ease this problem, the 
NASA Technical Standards Program developed the 
technical categorization taxonomy (Figure L). These 
categories are based on those developed by the 
Department of Defense and are the basis for both the 
NASA technical standards and the lessons learned 
categorization. In addition to enhancing the search 
capability by the end user of the System, this 
classification enables the user to more readily locate 
both standards and lessons learned applicable to the 
discipline of interest. 

During the life cycle of a program, the use of lessons 
learned and standards occur at the various appropriate 
phases to maximize the use of technology (Figure 2.). 
This technology is then reviewed by the managers and 
technical experts to determine the best direction of the 
program. As a result of technical decisions and 
accomplishments at the various phases of the life cycle, 
new lessons learned can then be identified and 
documented and related to technical standards for use 
on future program developments. Thus, the experience 
and knowledge gained can be readily made available to 
all concerned. 


Discipline Category 

Title 


Documentation and Configuration Management, Program Management 

1000 

Systems Engineering and Integration, Aerospace Environments, Celestial Mechanics 

2000 

Computer Systems, Software, Information Systems 

3000 

Human Factors and Health 

4000 

Electrical Systems, Electronics, Avionics/Control Systems, Optics 

5000 

Structures/Mechanical Systems, Fluid Dynamics, Thermal, Propulsion, Aerodynamics 

6000 

Materials and Processes, Parts 

7000 

System Test, Analysis, Modeling, Evaluation 

8000 

Safety, Quality, Reliability, Maintainability 

9000 

Operations, Command, Control, Telemetry/Data Systems, Communications 


Figure L Technical Categorization Taxonomy 
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Figure 2. Infusion into Programs and Projects 


Importance and Explanation of Linking of Lessons 
Learned with Standards 

At a glance it may appear that the linking of lessons 
learned to standards is an easily achievable task. After 
reviewing the data and resources during the pilot effort, 
it became clear that a major amount of time and effort 
is required to ensure a qualified product. The initial 
effort requires the location of lessons learned, whether 
electronic or paper form. Once this is accomplished, a 
detailed evaluation of the lesson learned must be 
performed to determine the validity of the lesson in 
relation to aerospace engineering applications. This is 
an important aspect of the undertaking relative to 
success of the endeavor. After reading the lesson, a 
search of related technical standards must be 
accomplished. This entails a detailed screening of 
technical standards that must be performed using 
specific keywords based on the content retrieved by the 
evaluation of the lesson. This also requires a thorough 
technical review. The use of advanced search engine 
capabilities will enable the reviewer to achieve better 
results in this search and in a more efficient manner. 
After reviewing the summary of the lessons learned 
evaluation results, the reviewer must then physically 
read the standard relative to the lesson learned 
evaluation to ensure the validity of the match. An 
independent reviewer should then verify the results. 
After completion of this task, the applicable lessons 
learned are linked with the standards in the database, 
ready for the next user of the System. 


Lessons Learned/Best Practices/Applications Notes-- 
Standards Integration System 

The NASA’s Lesson Learned/Best Practice/ 
Application Notes—Technical Standards Integration 
System has its origins in the initial technical standards 
program requirements that were established by the 
NASA Chief Engineer. The purpose of this System is 
to make available to those users within the <nasa.gov> 
domain the on-line technical standards linked with 
lessons learned. The objective is to provide ready 
access to the most recent technological advances within 
the aerospace industry. 

The Document Summary Page on the NASA Technical 
Standards Program Web site contains the pertinent 
information about the particular technical standards 
document (Figure 3.). The user is provided the title, 
current revision, status, and an abbreviated scope of the 
standards document. If there is an application note for 
the particular document, it is also included. An 
application note is a brief statement submitted by a user 
that briefly clarifies or limits the scope, use, or context 
of a given technical standard. Also contained on this 
Document Summary Page are lessons learned that have 
been screened and linked to the particular standard 
where relevance between the lesson and standard has 
been determined. 
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MIL-STD-1686 
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Revision: C 10/26/1995 

No of NASA Accesses since 06/2001 


148 


Status: Active 
SDO: MIL 

TITLE: ELECTROSTATIC DISCHARGE CONTROL PROGRAM FOR PROTECTION OF ELECTRICAL AND ELECTRONIC 
PARTS, ASSEMBLIES AND EQUIPMENT (EXCLUDING ELECTRICALLY INITIATED EXPLOSIVE DEVICES) (SUPERSEDING 

MIL S T D 1686B) 


NASA Status: Preferred 
Year Reaffirmed 


RaqL^ Standard Update ft o*fcabon] 


Base 


Date 10/25/1995 


19 pages 




Document Scope 


- 10 / 25 / 1995 ] 

The purpose of this standard is to establish comprehensive requirements for an ESD control program to minimize the effects of ESD on parts, assemblies, and equipment. 
An effective ESD control program will increase reliability and decrease both 

maintenance actions and lifetime costs. This standard shall be tailored for vanous types of acquisitions 



JPL 


4/26/2001 Requires that each facility have a document that describes how they implement ESD controls (for example, see 
MSEC -ROMT-2918) 
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Figure 3 . Document Summary Page Example 


Example of a Lessons Learned Influencing 
the Revision of a Standard 

As part of the flight certification of the High Energy 
Spectroscopic Imager (HESSI) spacecraft, a series of 
vibration tests at the NASA Jet Propulsion Laboratory 
(JPL) were conducted. The structural qualification test, 
a sine-burst test on a shaker table, subjected the 
spacecraft to a major overtest condition that resulted in 
significant structural damage to the spacecraft. 

The root cause of the overtest condition was the 
mechanical binding ("suction" or static friction) 
between the slip table and the granite mass. It resulted 
from physical contact between a portion of the slip 
table and the granite mass caused by a mechanical 
failure in the shaker's support structure. The stiction 
caused the shaker system to present highly non-linear 
gain characteristics to the control system making it 
impossible for the controller to calculate an appropriate 
forcing function. 


Other contributing factors identified were the lack of 
facility validation testing, age of the equipment, and test 
personnel lacking adequate knowledge of the test 
facility and it’s systems. Even though there are 
inherent risks associated with vibration testing, if these 
factors had been adequately addressed, then the failure 
may not have occurred. As a result of this test and the 
investigation that followed, plus similar vibration tests 
performed for flight certification of flight hardware, 
NASA-STD-7004, “Force Limited Vibration Testing” 
was revised based on the “lessons learned” 

Metrics and Feedback 

Metrics play a vital role in providing a true picture of 
the usage of the information on the NASA Technical 
Standards Program Web site. NASA tracks a variety 
of usage metrics for both technical standards and 
lessons learned. The standards and lessons learned 
major metrics tracked are the number of accesses and 
the Top 20 downloads. These are used to gauge the 
effectiveness of the Web site and also to identify where 
trends occur. This enables actions to be taken to 
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modify the Program content, etc. An example of the 
Top 20 Lessons Learned data sources accessed since 
July 2000 is shown in Figure 4. 

Feedback from the user is a very important feature of 
the NASA Technical Standards Program Web site. Not 
only does it enable improvements to be made on 
content of the Web site, but it also provides information 
on its operation. Change requests, complements, 
observations and questions, and suggestions and 
comments are routinely entered into the feedback 
system for a response to the user within 24 hours. This 
service helps ensure that all users remain comfortable 
with the performance of the Web site and get the 
information required to perform their tasks. Since May 
2001, 66 change requests, 49 complements, 140 
observations, 904 questions, 271 suggestions and 
comments were received. Incorporation of suggestions 
and comments has significantly enhanced the 
Program’s content and utility. Essentially all of the 
questions occurred in the early implementation stages 
of the Web site and were associated with the institution 
of a new “culture” for standards access and use. One of 
the more recent feedbacks was “ Regarding your 
questionnaire on the usage of the NASA Technical 
Standards Program, / have wanted to state for some 


time that the program has been an invaluable tool to me 
in all phases of my job which include almost all 
categories in the questionnaire. The NASA Technical 
Standards Program has greatly facilitated my work by 
providing easy access to applicable standards , 
specifications, and reports . / have recommended the 
Program to several colleagues who have also 
benefited. ” 

A pilot effort was undertaken in June 2003 to help 
identify the primary usage being made of standards 
accessed by users within the <nasa.gov> domain. This 
was done by requesting inputs from those logging on 
the NASA Technical Standards Program Web site to 
access full-text standards (Figure 5.). This revealed that 
the primary uses for the standards products, within the 
<nasa.gov> domain are in-house research and 
development activities and development of 
requirements for programs/project development. This 
pilot study is still underway and will be expanded, thus 
providing not only the NASA Technical Standards 
Program, but managers of the Agency’s programs and 
projects, senior Center and Headquarters managers, etc. 
with a better appreciation of the importance and use 
being made of technical standards products. 


Data Source 

Accesses 

A History of Aerospace Problems, Their Solutions, Their Lessons 

406 

System Engineering “Toolbox” for Design-Oriented Engineers 

120 

NASA Lessons Learned Information System 

119 

NASA Preferred Reliability Practices for Design and Test 

115 

Flight Project Lessons Learned Database 

97 

Space Engineering Lessons Learned 

90 

Systems Engineering Office Lessons Learned 

88 

Skyfab Lessons Learned 

88 

Software Engineering, Doing Requirements Right the First Time! 

80 

Technical Design Review Practices 

70 

Fastener Torque and Clamp Force Lessons Learned 

67 

Launch Vehicle Design Process: Characterization, Technical Integration, and Lessons Learned 

65 

Interplanetary Mission Design Handbook 

56 

Working on the Boundaries: Philosophies and Practices of the Design Process 

55 

Best Manufacturing Practices 

54 

"Non-Magnetic" Materials May Become Magnetized 

53 

COTS-Based Software Development: Processes and Open Issues 

51 

The Apollo 13 Accident 

46 

NASA Electronic Parts and Packaging 

44 

Design Guidelines for Assessing and Controlling Spacecraft Charging Effects Spacecraft Charging 
Effects 

42 


Figure 4. Top 20 Lessons Learned/Best Practices Accessed Via The 
NASA Technical Standards Program Web site 
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Design 

and Development 
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Acquisition 
of Parts or 
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of 
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Education 
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Other 
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23.3% 

29% 

18.6% 

11.5% 

3.7% 

8.8% 

5.1% 


Figure 5. Standards Use Questionnaire Results (June-August 2003) 


Some Key Issues—An Overview 
Lack of Common Format 

One of the key issues identified regarding the lessons 
learned data sources is the lack of a common format. A 
majority of specific lessons learned that have been 
linked to standards were obtained from the NASA 
Lessons Learned Information System 
http://llis.nasa.mn-7 . These lessons learned are 
presented in a common format that contains title, 
description and lesson learned information sections. 
Lessons found in other data sources can be in the form 
of program reports, failure analyses, technical 
memorandums, technical journal articles, or just a 
listing of programmatic or technical lessons learned. 
Therefore, the user must be flexible when searching and 
reviewing lessons learned data sources linked to 
standards, or otherwise accessed. 

Validation of Lessons Learned 

Before being linked to a technical standard, the lesson 
must be validated. This takes the effort of dedicated 
technical engineers to evaluate the lesson in detail and 
determine if it in fact is worthy to be used as a validated 
lesson learned. Since the technical community is so 
diverse, interpretation of the lesson can take many 
paths. It is critical that much attention be given to 
engaging technical reviewers having the right talents 
and engineering experiences. For the best evaluation, a 
technical working group reviewing the lessons is the 
preferred method to achieve the most viable final 
results. After the review is completed, the lesson can 
be integrated into the System and linked to a relevant 
technical standard. 


Accessibility 

In order to make the most of the integration of lessons 
learned with technical standards and infuse them into 
the programs and projects, it is preferred that they be 
identified as part of one or more phases of the program 
life cycle. Some, if not most, standards required should 
be determined at the initial program requirements 
review. Then at subsequent design and development 
phases, other standards should be identified for use, 
with control gates installed within the planning to 
ensure that the lessons and technical standards are 
addressed and used to ensure a successful mission or 
program. This will involve a change of culture in the 
technical community, but it will be one of high value to 
the program. 

Acquisition of Lesson Learned 

Accessing lessons learned is another obstacle that must 
be overcome. Some lessons may be found 
electronically over the Internet, and are readily 
accessible. Others may be in paper form stored in filing 
cabinets, located on password protected Web sites, or 
proprietary. In these cases, additional steps are required 
if they are to be accessible to the user. 

Awareness of Data Sources and Content 

To remain current with the available technology, as 
new lessons are developed and made available, the user 
must be informed. It is therefore imperative that 
announcements, such as “What’s New” on the NASA 
Technical Standards Program Web site or emails, be 
sent to users who register for notification requests 
within their discipline category. The user must be made 
aware of the addition of current lessons learned in a 
timely and user-friendly manner in order to be able to 
take advantage of current technology. Development of 
this initiative is currently in the planning stages as part 
of the pilot program. 
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